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. RESEARCH & PRACTICE IN ASSESSMENT 

Abstract 

In the new era of big educational data, learning analytics (LA) offer the 
possibility of implementing real-time assessment and feedback systems 
and processes at scale that are focused on improvement of learning, 
development of self-regulated learning skills, and student success. How¬ 
ever, to realize this promise, the necessary shifts in the culture, techno¬ 
logical infrastructure, and teaching practices of higher education, from 
assessment-for-accountability to assessment-for-learning, cannot be 
achieved through piecemeal implementation of new tools. We propose 
here that the challenge of successful institutional change for learning 
analytics implementation is a wicked problem that calls for new adap¬ 
tive forms of leadership, collaboration, policy development and strategic 
planning. Higher education institutions are best viewed as complex sys¬ 
tems underpinned by policy, and we introduce two policy and planning 
frameworks developed for complex systems that may offer institutional 
teams practical guidance in their project of optimizing their educational 

systems with learning analytics. 


Embracing Big Data in Complex 
Educational Systems: The Learning Analytics 
Imperative and the Policy Challenge 


Jn education, we are awash in data about our learners and educators, our 
technologies and activities, achievements and performance. To date these data have 
rarely been mined intelligently with the goal of improving learning and informing teaching 
practice, although evidence from other sectors such as marketing, sports, retail, health 
and technology suggests that the effective use of big data can offer the education sector 
the potential to enhance its systems and outcomes (Manyika et al., 2011). Norris and Baer 
(2013) have noted that, “Data expands the capacity and ability of organizations to make 
sense of complex environments” (p. 13). In this context, learning analytics (LA) offers 
the capacity to investigate the rising tide of learner data with the goal of understanding 
the activities and behaviors associated with effective learning, and to leverage this 
knowledge in optimizing our educational systems (Bienkowski, Feng, & Means, 2012; 
Campbell, DeBlois, & Oblinger, 2007). Indeed, in a world of larger and larger data sets, 
increasing populations of increasingly diverse learners, constrained education budgets 
and greater focus on quality and accountability (Macfadven & Dawson, 2012), some argue 
LUKKhorUjM/MLh that using analytics to optimize learning environments is no longer an option but an 

imperative. The value of such analytics is highlighted by the authors of the McKinsey 
Email Global Institute (Manyika et ah, 2011) noting that, “In a big data world, a competitor that 
leah.macfadyen@ubc.ca fails to sufficiently develop its capabilities will be left behind...Early movers that secure 

access to the data necessary to create value are likely to reap the most benefit” (p. 6). 
Education can no longer afford not to use learning analytics. As Slade and Prinsloo (2013) 
maintain, “Ignoring information that might actively help to pursue an institution’s goals 
seems shortsighted to the extreme” (p. 1521). 
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In this article we consider ways in which learning analytics can support and contribute 
to the development of new approaches to the assessment of learning, and the degree to 
which new adaptive policy and planning approaches will be needed to bring about the kind 
of institutional change such proposals demand. We emphasize that successful institutional 
adoption demands comprehensive development and implementation of policies to address 
LA challenges of learning design, leadership, institutional culture, data access and security, 
data privacy and ethical dilemmas, technology infrastructure, and a demonstrable gap in 
institutional LA skills and capacity (Siemens, Dawson, & Lynch, 2013). Moreover, we take the 
position that educational institutions are complex adaptive systems (Gupta & Anish, 2009; 
MacLennan, 2007; Mitleton-Kelly, 2003), and therefore that simplistic approaches to policy 
development are doomed to fail. Instead, we will introduce strategy and policy frameworks and 
approaches developed for complex systems, including frameworks that offer the potential to 
identify points of intervention (Corvalan, Kjellstrom, & Smith, 1999), with the goal of offering 
educational institutions practical guidance. 

Assessment Practices: A Wicked Problem in a Complex System 

There is no better exemplar in higher education than assessment to demonstrate 
how institutional policy can impact practice both positively and negatively. The practice 
of assessment has for some time been mired in debate over its role as either a measure of 
accountability or a process for learning improvement. While the majority of education 
practitioners lean towards assessment as a process for improving student learning, assessment 
nonetheless remains firmly positioned as an important tool for determining accountability 
and demonstrating quality. As McDonnell (1994) previously argued, assessment policies 
function as a mechanism to provide government with a high level of influence over classroom 
practice. In essence, assessment acts as a powerful tool to manage aspects of learning and 
teaching. It is not surprising, then, that assessment policy has numerous invested stakeholders 
- learners, educators, administrators and government - all vying for a larger stake in the game. 
The diversity of stakeholders, priorities, outcomes and needs make any substantial change to 
assessment policy and practice a considerable challenge to say the least. 

Assessment practice will continue to be intricately intertwined both with learning 
and with program accreditation and accountability measures. Such interconnectedness in 
educational systems means that narrow efforts to implement changes in policy and practice 
in one area (for example, by introducing new approaches to tracking and measuring learning) 
may have unanticipated consequences elsewhere in the system. For example, the US 
education policy No Child Left Behind drastically reshaped not only the testing processes 
employed to identify poor literacy and numeracy standards, but also affected what was taught 
and how it was taught. Jacob (2005) documented the unintentional outcomes of this new 
accountability policy classroom practice, noting, for example that such high-stakes testing 
encouraged teachers to steer low-performing students away from subjects that were included 
in the accountability program. While the ethos of the policy had some merit in attempting 
to address declining numeracy and literacy skills in the US, the associated incentives and 
measures resulted in crossed performance indicators. Dworkin (2005) also expands on this 
point, noting that teacher promotion standards were linked to class performance in the high 
stakes tests. This practice essentially encouraged teachers to narrow the curriculum and teach 
to the test, beautifully illustrating Goodhart’s Law, which states that when a measure becomes 
a target it ceases to be a useful measure (Elton, 2004). 

In the complex systems of higher education, current performance assessment and 
accountability policies may be the forces driving (Corvalan et al., 1999) the continued focus 
on high-stakes snapshot testing as a means of producing comparative institutional data, in 
spite of the well-articulated weakness of such an approach for understanding student learning. 
The continuing primary use of grades in determining entry to university, the Australian 
Government’s National Assessment Plan for Literacy and Numeracy (NAPLAN) 1 measures, 
the OECD’s Programme for International Student Assessment (PISA) 2 and similar programs, 
demonstrate that there is much invested in the retention of these measures for benchmarking 
individuals, schools, districts, states and countries. Wall, Uursh and Rodgers (2014) have 

1 http://education.qld.gov.au/naplan/ 

2 http:/www.oecd.org/pisa/ 
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argued, on the other hand, that the perception that students, parents and educational leaders 
can only obtain useful comparative information about learning from systematized assessment 
is a false one. Instead, alternate complementary assessment practices - practices that make 
better use of the rich array of educational data now available - may well offer more effective 
approaches to improving learning, especially processes that reveal development of student 
understanding over time (Wiliam, 2010). 

In his criticism of assessment practices, Angelo (1999) suggested that as educators 
we must emphasize assessment as a means for improving student learning rather than a 
mechanistic, technical process used to monitor performance. lie argued that assessing for 
learning necessitates a focus on developing practices that help the educator and learner gather 
evidence of learning progress, rather than on identifying the students that produce the “right” 
or “wrong” answers. The importance of developing better formative or embedded assessment 
models has also been reiterated by the OECD Innovative learning environments project 
(Dumont, Istance, & Benavides, 2010) and educational researchers have similarly illuminated 
that regular feedback at the process level is more effective for enhancing deeper learning (for 
review, see Hattie & Timperley, 2007). 

Despite the widespread recognition of the need for a more effective assessment 
paradigm, implementation is a challenge, and calls for development of new policies and 
implementation strategies directed at improving accountability for learning though 
practices driven by learning. Differentiating assessment-for-learning from assessment-for- 
accountability within the educational system forms part of the wicked problem of institutional 
change in higher education that we seek to explore here. As with all complex systems, even 
a subtle change may be perceived as difficult, and be resisted (Head & Alford, 2013). For 
example, under normal classroom circumstances the use of assessment at the process level 
for improving learning requires substantial and sustained engagement between the educator 
and students and can be an extremely time intensive process. Implementing such time 
intensive assessment models for large (and growing) university classes is not feasible, and 
typically scalable models of assessment such as multiple choice exams are implemented 
instead. It is unrealistic to consider that educators will adopt time-consuming longitudinal 
and personalized assessment models given the massive increase in resources and workload 
that would be required. 

Learning Analytics and Assessment-for-Learning 

A wide range of authors in this special issue illustrate ways in which learning 
analytics - which comes with its own set of implementation challenges and hurdles - 
has the potential to provide learners with sustained, substantial and timely feedback to 
aid understanding and improve student learning skills, while circumventing the challenge of 
educator workload. We also offer a discussion of how learning analytics may support development 
of self-regulated learning in Box 1, inset. Analytics can add distinct value to teaching and 
learning practice by providing greater insight into the student learning process to identify the 
impact of currictdum and learning strategies, while at the same time facilitating individual 
learner progress. Nor does the adoption of learning analytics preclude traditional or alternate 
assessment practices that may be required by accreditation and accountability policies. While 
current assessment policy may be driven by conflicting intentions - accountability and quality 
assurance requirements versus promotion of student learning - learning analytics can meet 
both. More simply put, LA addresses the need for quality assurance and learning improvement. 

Technological Components of the Educational System 
and Support of Learning Analytics 

The LA-supported approaches to assessment of learning envisioned in this 
article - indeed, in this entire edition - assumes a technological layer that is capable of 
capturing, storing, managing, visualizing and processing big educational data - the millions 
of events occurring in diverse learning scenarios and platforms. Transformation of assessment 
practices to embrace and integrate learning analytics tools and strategies in support of 
teaching and learning therefore demands effective institutional technology infrastructures. 
The production of data in every technology-mediated interaction occurring in a learning 
environment, the need for more effective provision of feedback, and the need for more 
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Box 1 

Learning Analytics for Assessing Student Learning 


Differentiating assess¬ 
ment—for—learning from 
assessment-for-account- 
ability within the educa¬ 
tional system forms part 
of the wicked problem 
of institutional change in 
higher education that we 
seek to explore here. 


Provision (to learners and educators) of automated analytics that provide feedback 
on learner study behaviors, progress and outcomes will not only enhance academic 
performance but also develop student self-regulated learning (SRL) skills, and SRL 
proficiency has been demonstrated to be a significant predictor of academic success 
(e.g., Butler & Winne, 1995; Pintrich, 1999; Zimmerman, 2002). Student motivation 
and capacity to undertake accurate self-monitoring had a direct impact on the level 
and quality of their study and therefore, their overall learning progression and academic 
achievement (Dunlosky & Thiede, 1998). Conversely, poor performers are poor at 
evaluating their own ability or judging their own learning skills (Kruger & Dunning, 

1999). For these reasons, it is argued that a core goal of any effective pedagogical 
strategy must include the development of student meta-cognitive skills or judgment of 
(own) learning (JOL). Feedback on assessment is one key approach that is often adopted 
to assist students in developing meta-cognitive skills, but because provision of relevant 
feedback can be labor-intensive, it is often delayed and provided at a time when it is no 
longer useful to the student to aid their learning. 

Recent research posits that SRL is a process of temporal events that evolve during 
learning (Azevedo & Aleven, 2013). This research, alongside recent developments in 
learning analytics, data mining and machine learning is providing new methods for 
developing novel insights into student learning processes. Historically, assessment 
and development of student SRL has made use of tasks associated with JOL which 
generally involve asking a student to assess how effectively they have understood a 
particular concept (Dunlosky & Lipko, 2007). This self-reported rating is then correlated 
against their overall test performance to gain insight into the student’s meta-cognitive 
proficiency. While JOL has commonly relied on self-report methodologies such as think 
aloud protocols and surveys, these have inherent limitations such as poor recall, and 
biased responses (Richardson, 2004). 

New options for assessing student learning behaviors are emerging as a result of advances 
in learning analytics and natural language processing (NLP), and alternate models have 
sought to capture actual learner behavior (in lieu of self-reports) from interactions with 
technology-based learning activities. For example, oft-cited SRL researcher Phil Winne 
has previously reported that student online interaction data can provide significant 
indicators of SRL proficiency (e.g., Winne, 2010; Zhou & Winne, 2012). Winne has 
developed the software application nStudy as a web tool that can collect very fine 
grained, time stamped data about individual learner interactions with online study 
materials. The trace data is then used to provide insight and feedback into the learner’s 
cognitive choices as they interact with the online information. Essentially, the tool 
makes data for reflection available to both the individual learner and the educator. 


comprehensive formative and summative assessment translates into a rich set of requirements 
of the current technological infrastructures. Although learning management systems (LMSs) 
still host a large percentage of technology-mediated educational activities, educational 
institutions are recognizing the need to re-assess the concept of teaching and learning space 
to encompass both physical and virtual locations, and adapt learning experiences to this new 
context (Thomas, 2010). Thus, together with the need for cultural change and a focus on 
pedagogical relevance, an additional sociotechnical factor critical to the adoption of learning 
analytics is technology itself (Box 2). 
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The evolution of technology in recent years offers an unprecedented capacity to store 
large data sets, and applications using big data are well established in contexts such as business 
intelligence, marketing and scientific research (Dillon, Wu, & Chang, 2010). Education faces a 
particular challenge that derives from the rich variety of technological affordances emerging in 
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learning environments. From an LMS-centric approach consolidated in the early 2000s, we are 
now entering an era in which learning may occur anywhere, at any time, with multiple devices, 
over a highly heterogeneous collection of resources, and through multiple types of interactions. In 
this new scenario, learning analytics tools need to comply with requirements in the following areas: 

1. Diverse and flexible data collection schemes: Tools need to adapt to increasing 
data sources, distributed in location, different in scope, and hosted in any platform. 

2. Simple connection with institutional objectives at different levels: 
information needs to be understood by stakeholders with no extra effort. Upper 
management needs insight connected with different organizational aspects 
than an educator. User-guided design is of the utmost importance in this area. 

3. Simple deployment of effective interventions, and an integrated and 
sustained overall refinement procedure allowing reflection. 

From the technological point of view, learning analytics is an emerging discipline 
(Siemens, 2013) and its connection with assessment remains largely unexplored (Ellis, 2013). 
This situation is even more extreme when considering the assessment of competences and 
learning dispositions (Buckingham Shum, 2012). Educational institutions need technological 

Box 2 

Sociotechnieal Infrastructure Needs for Effective Learning Analytics 


Several initiatives are already tackling the problem of flexible data collection schemes. 
For example the ADL Experience API 3 released in 2013 has been proposed as a solution 
that can promote interoperability between data collected in different environments 
and platforms. The interface offers the possibility of capturing a wide variety of 
events in experiences with heterogeneous scenarios (Glahn, 2013). Similarly, the 
IMS Global Consortium has proposed that the Learning Measurement Framework 
IMS Caliper 4 - containing descriptions to represent metrics, sensor API and learning 
events - will facilitate the representation and processing of big data in the learning 
field. In parallel, the concept of a Learning Record Store (LRS) has been proposed as 
a framework for storing and manipulating data from distributed events in a learning 
environment, encoding not only interaction among stakeholders, but among resources. 
This information is then made available through a service-based interface to other 
systems within an institution (or across multiple institutions) for further analysis and 
processing. 

Numerous attempts have been made to meet diverse stakeholder reporting and data 
access needs by production of so-called dashboards that show a canvas of multiple 
visualizations. Common limitation of these graphical representations, however, are their 
actual utility and usability (Verbert, Duval, Klerkx, Govaerts, & Santos, 2013). Adapting 
presentation of information to user context, needs and interests is another important 
factor that must be taken into account if we wish to facilitate the uptake of learning 
analytics solutions. 

The third requirement for technology supporting learning analytics is that it can 
facilitate the deployment of so-called interventions, where intervention may mean any 
change or personalization introduced in the environment to support student success, 
and its relevance with respect to the context. This context may range from generic 
institutional policies, to pedagogical strategy in a course. Interventions at the level of 
institution have been already studied and deployed to address retention, attrition or 
graduation rate problems (Ferguson, 2012; Fritz, 2011; Tanes, Arnold, King, & Renmet, 
2011). More comprehensive frameworks that widen the scope of interventions and 
adopt a more formal approach have been recently proposed, but much research is still 
needed in this area (Wise, 2014). 
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3 http://www.adlnet.gov/tla 

4 http:/www.imsglobal.org/IMSLearningAnalyticsWP.pdf 
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solutions that are deployed in a context of continuous change, with an increasing variety of data 
sources, that convey the advantages in a simple way to stakeholders, and allow a connection 
with the underpinning pedagogical strategies. 

In turn, these technological requirements point to a number of other critical contextual 
factors that must form part of any meaningful policy and planning framework for employing 
learning analytics in service of improved assessment. Foremost among these is the question 
of access to data, which needs must be widespread and open. Careful policy development is 
also necessary to ensure that assessment and analytics plans reflect the institution’s vision 
for teaching and strategic needs (and are not simply being embraced in a panic to be seen to 
be doing something with data), and that LA tools and approaches are embraced as a means of 
engaging stakeholders in discussion and facilitating change rather than as tools for measuring 
performance or the status quo. 


The Challenge: Bringing about Institutional Change in Complex Systems 

While the vision of improving student learning and assessment through implementation 
of effective learning analytics tools and approaches holds promise, the real challenges of 
implementation are significant. In this article we have identified only two of the several critical 
and interconnected socio-technical domains that need to be addressed by comprehensive 
institutional policy and strategic planning to introduce such attractive new systems: the 
challenge of influencing stakeholder understanding of assessment in education, and the 
challenge of developing the necessary institutional technological infrastructure to support 
the undertaking. Meanwhile, of course, any such changes must coexist with the institution’s 
business as usual obligations (Head & Alford, 2013). 


Transformation of 
assessment practices to 
embrace and integrate 
learning analytics tools 
and strategies in support 
of teaching and learning 
therefore demands effec¬ 
tive institutional technol¬ 
ogy infrastructures 


It may not be surprising, then, that globally, education lags behind all other sectors in 
harnessing the power of analytics. A preliminary analysis indicates that educational institutions 
simply lack the practical, technical and financial capacity to effectively gather, manage and 
mine big data (Manyika et al., 2011). As Bichsel (2012) notes, much concern revolves around 
“the perceived need for expensive tools or data collection methods” (p. 3). Certainly, evidence 
suggests that data access and management are proving to be significant hurdles for many 
institutions. The first survey of analytics implementation in US higher education in 2005 found 
that of 380 institutions, 70% were at Stage 1 of a five-stage implementation process: “Extraction 
and reporting of transaction-level data” (Goldstein & Katz, 2005). Four years later, a study 
of 305 US institutions found that 58% continued to wrangle data in Stage 1, while only 20% 
reported progress to Stage 2: “Analysis and monitoring of operational performance” (Yanosky, 
2009). More recently, investigators have reported that while some 70% of surveyed institutions 
agreed that analytics is a major priority for their school, the majority of respondents suggested 
that data issues (quality, ownership, access, and standardization) were considerable barriers to 
analytics implementation, and as such most were yet to make progress beyond basic reporting 
(Bichsel, 2012; Norris & Baer, 2013). 


To further unpack the complexities of analytics adoption a growing number of 
commentators are exploring the more nuanced sociotechnical factors that are the most likely 
barriers to institutional LA implementation. For instance, elements of institutional “culture, 
capacity and behavior” (Norris, Baer, Leonard, Pugliese, & Lefrere, 2008). There is recognition 
that even where technological competence and data exist, simple presentation of the facts (the 
potential power of analytics), no matter how accurate and authoritative, may not be enough to 
overcome institutional resistance (Macfadyen & Dawson, 2012; Young & Mendizabal, 2009). 
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Why Policy Matters for Learning Analytics 

Higher education institutions are a superb example of complex adaptive systems 
(CASs) (Cilliers, 1998; Gupta & Anish, 2009; MacLennan, 2007; Mitleton-Kelly, 2003) and 
exist in a state that some have characterized as organized anarchy (Cohen & Marsh, 1986). 
Together with institutional history and differences in stakeholder perspectives (Kingdon, 1995; 
Sabatier, 2007), policies are the critical driving forces that underpin complex and systemic 
institutional problems (Corvalan et al., 1999) and that shape perceptions of the nature of 
the problem(s) and acceptable solutions. Below, we argue that it is therefore only through 
implementation of planning processes driven by new policies that institutional change can 
come about. 
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The challenge of bringing about institution-wide change in such complex and 
anarchic adaptive systems may rightly be characterized as a “wicked problem”- a problem 
that is “complex, unpredictable, open ended, or intractable” (Churchman, 1967; Head & 
Alford, 2013; Rittel & Webber, 1973). Like all complex systems, educational systems are 
very stable, and resistant to change. They are resilient in the face of perturbation, and exist 
far from equilibrium, requiring a constant input of energy to maintain system organization 
(see Capra, 1996). As a result, and in spite of being organizations whose business is research 
and education, simple provision of new information to leaders and stakeholders is typically 
insufficient to bring about systemic institutional change. One factor hindering institutional 
change for better use of analytics by educational institutions appears to be their “lack of 
data-driven mind-set and available data” (Manyika et al., 2011, p. 9). Interestingly, this 
observation is not new, and was reported with dismay in 1979 by McIntosh, in her discussion 
of the failure of institutional research to inform institutional change. Ferguson et al. (in press) 
reprise McIntosh’s arguments in relation to learning analytics, suggesting that additional 
barriers to adoption include academics’ unwillingness to act on findings from other disciplines; 
disagreement over the relative merits of qualitative vs. quantitative approaches to educational 
research; a tendency to base decisions on anecdote; the reality that researchers and decision 
makers speak different languages; lack of familiarity with statistical methods; a failure to 
effectively present and explain data to decision makers; and the reality that researchers tend to 
hedge and qualify conclusions. Norris and Baer (2013) meanwhile note that the analytics IQ, of 
institutional leaders is typically not high, precluding effective planning. In other words, a range 
of political, social, cultural and technical norms shape educational systems and contribute to 
their stability and resistance to change. 

Elsewhere, we reported on a case study failure of learning analytics to inform 
institutional planning (Macfadyen & Dawson, 2012), and noted that the culture of educational 
institutions has historically valorized educator/faculty autonomy and resisted any administrative 
efforts perceived to interfere with teaching and learning practice. We proposed that in order 
to overcome institutional resistance to innovation and change driven by learning analytics, 
educational institutions urgently need to implement planning processes that create conditions 
that allow stakeholders across the institution to both think and feel positively about change - 
conditions that appeal to both the heart and the head. 

Social marketing theorists (Kotler & Zaltman, 1971) and change management experts 
(Kavanagh & Ashkanasy, 2006; Kotter, 1996) similarly argue that social and cultural change 
(that is, change in habits, practices and behaviors) is not brought about by simply giving 
people large volumes of logical data (Kotter & Cohen, 2002). Social theorists have argued 
that since value perspectives ground the major social issues of modern life, scientific analyses 
and technical rationality are insufficient mechanisms for understanding and solving complex 
problems (Head & Alford, 2013; Rein, 1976; Schon & Rein, 1994). Instead, what is needed are 
comprehensive policy and planning frameworks to address not simply the perceived shortfalls 
in technological tools and data management, but the cultural and capacity gaps that are the 
true strategic issues (Norris & Baer, 2013). 

Policy and Planning Approaches for Wicked Problems in Complex Systems 

Policies are, simply, principles developed to guide subjective and/or objective decision 
making, with the goal of achieving rational and desirable outcomes. They are statements of 
intent that capture organizational goals, and are typically implemented via planned procedures 
or protocols. A large and established literature on policy development already exists in fields 
such as political science and business, from which have emerged a range of classical policy 
cycle tools and heuristics that have been highly influential (Nakamura, 1987). Contemporary 
critics from the planning and design fields argue, however, that these classic, top-down, 
expert-driven (and mostly corporate) policy and planning models are based on a poor and 
homogenous representation of social systems mismatched with our contemporary pluralistic 
societies, and that implementation of such simplistic policy and planning models undermines 
chances of success (for review, see Head & Alford, 2013). Importantly, they also insist that 
modern policy problems are not technical puzzles that can be solved through the application 
of scientific knowledge, but instead exist in continuous states of flux within dynamic systems 
and have communicative, political and institutional elements. Solutions to such ill-defined 
and multi-factorial challenges, they argue, will always be provisional, and must be negotiated 
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between multiple stakeholders in situations of ambiguity, uncertainty and values disagreement 
(Rittel & Webber, 1973). A number of theorists have also emphasized that solutions to wicked 
problems - actually complex systems of inter-related problems - “can seldom be obtained 
by independently solving each of the problems of which it is composed . . . Efforts to deal 
separately with such aspects of urban life as transportation, health, crime, and education seem 
to aggravate the total situation” (Ackoff, 1974, p. 21). 


From the technological 
point of view, learning 
analytics is an emerg¬ 
ing discipline and its 
connection with assess¬ 
ment remains largely 
unexplored. 


Systems theory offers two key areas of insight that are significant for pokey development 
for complex educational systems. First, systems theorists recognized that while systems - 
from a single atom to a universe - may appear to be wildly dissimilar, they are all governed by 
common patterns, behaviors and properties: their component parts are multiply interconnected 
by information flows, with identifiable and predictable feedbacks, inputs, outputs, controls and 
transformation processes; they are dynamic, differentiated and bounded; they are hierarchically 
organized and differentiated; and new properties can arise within them as a result of interactions 
between elements. Second, systems theory observes that systems tend to be stable, and that their 
interconnectedness facilitates resilience (for a review of systems theory, see Capra, 1996). 

These observations not only illuminate why piecemeal attempts to effect change in 
educational systems are typically ineffective, but also explains why no one-size-fits-ak prescriptive 
approach to policy and strategy development for educational change is available or even possible. 
Usable policy frameworks wik not be those which offer a to do list of, for example, steps in learning 
analytics implementation. Instead, successful frameworks will be those which guide leaders and 
participants in exploring and understanding the structures and many interrelationships within 
their own complex system, and identifying points where intervention in their own system wik be 
necessary in order to bring about change. 


Drawing on systems and complexity theory, a new generation of authors have begun 
to develop accounts of so-called adaptive approaches to policy and planning for complex 
systems which can allow institutions to respond flexibly to ever-changing social and 
institutional contexts and challenges (Berkhout, Leach, & Scoones, 2003; Haynes, 2003; 
Milliron, Malcolm, & Kil, 2014; Tiesman, van Buuren, & Gerrits, 2009; Young & Mendizabal, 
2009). A full review of adaptive management strategies is beyond the scope of this paper, and 
has been comprehensively undertaken by Head and Alford (2013), who highlight the critical 
roles of cross-institutional collaboration, new forms of leadership (moving beyond the 
orthodox model of transformational leadership) and the development of enabling structures 
and processes (for example, budgeting and finance systems, organizational structure, 
human resources management, and approaches to performance measurement and program 
evaluation). We offer here two sample policy and planning models that may offer valuable 
practical guidance for collaborative teams and leaders in higher education seeking to bring 
about systemic institutional change to support learning analytics. 
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Figure 1. The RAPID Outcome Mapping Approach (ROMA) 
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First, and as we have proposed elsewhere (Ferguson et al., in press) we offer a modification 
of Young and Mendizabal’s (2009) Rapid Outcome Mapping Approach (ROMA) model (Figure 1) 
as a policy and planning heuristic for learning analytics implementation. Originally developed 
to support policy and strategy processes in the complex field of international development, the 
seven-step ROMA model is focused on evidence-based policy change. It is designed to be used 
iteratively, and to allow refinement and adaptation of policy goals and the resulting strategic plans 
over time and as contexts change, emphasizing the provisional nature of any solutions arrived at. 
Importantly, the ROMA process begins with a systematic effort at mapping institutional context 
(for which these authors offer a range of tools and frameworks) - the people, political structures, 
policies, institutions and processes that may help or hinder change. This critical activity allows 
institutions to identify the key factors specific to their own context that may influence (positively 
or negatively) the implementation process, and therefore also has the potential to illuminate points 
of intervention and shape strategic planning. 



National and institutional Out-dated institutional 

policies on assessment technology policies 


1X1 


A culture of end-of-term Focus on reporting and 
test-based assessment data security 



Teaching/learning to the 
test 

1 

Limited in-time feedback 
on learning available to 
students 



♦ 

Data inaccessible to 
most educational 
stakeholders 

I 

No development of a 
culture of data use to 
improve learning 


i i 

Lost opportunities for learning interventions 



Figure 2. Cause-effect (DPSEEA) framework for institutional assessment and technology policies 
(modified from Corvalan et ah, 1999). 


Second, Corvalan et al.’s (1999) “cause-effect framework” (or DPSEEAframework) usefully 
assists in identifying the multiple linkages that may exist between the driving forces underpinning 
complex systems, illuminating the multiple points in a complex system of relationships where 
action may be needed to effect change. Such a framework can, they suggest, “be used to weigh 
alternatives and to design step-by-step programs for dealing with a particular.. .problem” (p. 659). 
Figure 2 offers a preliminary modification of this framework to represent institutional effects of, 
for example, technology and assessment policies, and may be a useful context mapping tool in the 
ROMA process. 

Use of these models for institutional LA policy development is only in the very early 
stages, although we have explored elsewhere (Ferguson et ah, in press) the ways in which a small 
number of apparently successful institutional LA policy and planning processes have pursued 
change management approaches that map well to such frameworks. In future work, we hope to 
be able to present more robust and critical review of real-time application of these frameworks in 
institutional planning, and their possible effectiveness or limitations. 

In the meantime, readers may review both frameworks and immediately dispute the 
stages, levels, linkages, effects or impacts in relation to their own institutional context. But this is, of 
course, the very point of such adaptive models, which can and should be disputed, negotiated and 
modified as needed for local institutional contexts, to guide relevant local action. To paraphrase 
Head and Alford (2013), when it comes to wicked problems in complex systems, there is no one- 
size-fits-all policy solution, and there is no plan that is not provisional. 

Rather, the more important role of such frameworks is to continuously remind us of the 
need for a holistic understanding of institutional context if the goal is institutional change, including 
external and internal influences, political and cultural context, the evidence itself, and the links: 


It may not be surprising, 
then, that globally, 
education lags behind 
all other sectors in 
harnessing the power of 
analytics. A preliminary 
analysis indicates that 
educational institutions 
simply lack the practical, 
technical and financial 
capacity to effectively 
gather, manage and mine 
big data. 
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“All of the other actors and mechanisms that affect how the evidence gets into the policy process” (Young 
& Mendizabal, 2009). They can assist in identifying points of intervention (Corvalan et al., 1999) in the 
complex adaptive system that is education, to offer leaders and practitioners additional insight and tools in 
their project of optimizing the system with learning analytics. 
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